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ABSTRACT 

The major aims of the "Annual Review of Information 
Science and Technology" are (1) to describe and appraise progress in 
information science and technology; (2) to provide a systematic, 
dependable tool that can relieve professionals from winnowing through 
a wealth of literature in their field each year; ?3) to direct the 
reader to people, projects, institutions and literature that may be 
helpful to them in their work; (4) to construct a permanent reference 
file in which the reader can trace the work of a person, a project, 
an institution, or a problem or topic of interest and (5> to 
introduce somewhat greater order into the field of information 
science and technology. (Author/MM) 
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heig;ht of the production cycle for Volume 6 of the Annual Review , and the 
time in which we begin to intensify our work for VoluiTiB 7 covering the 1971 
literature. In addition, I have just seen a very interesting and provocative 
peiper on the Annual Review vjritten by Dr. Tefko Saraceva c of Case Western 
Reserve Univei’sity. Tefko was kind enough to send me the paper and ask for 
my reactions. Although I disagree with, some of his conclusions , ha is a 
respected colleague, and his criticisms and suggestions deserve close 
attention. 

Pvecent Criticisms of the Annual Review 



Tefko’ s paper raises several major points of concern W'ith the Annual Review 
series. The first of these involves the charge that the app* roach of the 
reviewers, i.e., the chapter authors, is "uncritical” and that one cannot 
obtain an impression regarding the quality of the v/orks being revie\ced. 

1X^70 solutions are proposed. One is to make the approach to reviewing more 
critical and evaluative. The second is to ab c- :^sent format of 

the and to have, instead, a two-part publication. The first 

part would contain state-of-the-art monographs covering topics as a whole 
(rather than a one- or two-year slice) ; the second part would contain a 
critical atmotated bibliography of the year’s literature. 

Tlie second criticism is that there is too much emphasis on technology. It 
is argued that t he_„^Annaa3^ Re v_i does not distinguish between information 
science, information technology, and inf onaation practice. It is also 
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argued that there is excessive orientation toward technology in general and 
computers in particular and that the ^tonual Review is, in effect, still 
promoting technology in spite of its '^failures.” Three solutions are pro- 
posed. First, cut the emphasis on technology. Second, treat information 
science as a science and give theory a prominent review, when and where it 
exists. Third, pay more ' attention to social and philosophical issues in 
the field. 

The third major criticism is that the wrong literature is being reviewed. 

It is argued that there are too few citations to items from journals, and 
far too many citations to technical reports and other kinds of items printed 
without any editorial constraints. The statement is made: **Very few of 

these items are worthwhile, and these that are end up anyhow as journal 
articles." Two solutions are proposed. First, it is suggested that 
technical report literature be omi ~ ced, with very, very few exceptions. 
Brochures, public relation releases and the like should be omitted completely 
Papers presented at meetings that had no published proceedings should be 
omitted completely, since "these are completely inaccessible." The second 
suggestion is that "qualitative judgment" should be imposed upon the 
literature that is cited, giving priority to articles from journals having 
peer review procedures. News items, announcements, features from journals, 
if treated at all, should be held at an absolute minimum. 
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Tefko's review had other points worthy of mention, although there is not 
really time to discuss them. For example, the following criticisms were 
made : 

• the Annual Review is rather blind to the distinctions between 
things that are operational, in experimental use, or only figments 

in the minds of their conceivers* 

• The chapter authors are forced to construct their own topic 
organization. 

• The structure of the Annual Review content is based on the pattern of 
large grants as awarded by major Federal granting agencies in the 
I960's. 

• The structure is frozen, so one cannot discern whether an area 
is rapidly developing or declining. 

I don't want to give the impression that the paper was totally negative about 
the Annual Review. It was, in fact, quite complimentary on several aspects 
of the series and about my role in it. However, it is important to give 
particularly careful attention to the criticisms and suggestions for improve- 
ment. As I told Tefko, I am a firm believer not only in systematic evaluation 
but also in having such evalLiation done by capable- — but ei^sentially disinterested 
objective persons. Whatever else I am, I am not diijinterested in any matter 
relating to the Annual Review. So if I disagree with some of the criticisms 




and bite back somewhat, you may need to discount my views to some extent. 
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How one should react to the several criticisms I have reported depends on 
two things : 

• What one believes an annual review is for, i.e,, the impact it is 
intended to achieve. 

• Whether that intended impact is, in fact, being achieved. 

To help you think about the various criticisms and suggestions, iM lii:e to 
tell you something about the purposes of the Annual Review, the mechanics of 
production, and the impact of the series. 

Purposes and Mechanisms of the Annual Review 

Annual reviews are not new entities ; One organization has been publishing annual 
reviews for about 25 years and now publishes more than a dozen series. The 
plan, format, and content of the Annual Review were based on this history and 
were not invented from whole cloth. 

Few reviews are explicit about their purposeFJ. ^ ' general Is zo 

describe and appraise progress in the particular discipline concerned, for a 
particvilar period of time. The major purpose of the Annual Reviexi; is to describe 
and appraise progress in information science and technology- I also have had 
other, more detailed purposes, e.g,, 

. To provide a systematic, dependable tool that could relieve professionals 
in their field of the necessity of trying to read 3--6000 pieces of 
literature in their field each year. 
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. To direct our readers to people > projects, institutions., and 
• literature that might be helpful to them in. their work, 

• To construct an information base for our field, i,e., a permanent 
reference file in which one could trace, over time, the work of a 
person, a project, an institution, or a problem or topic of interest. 



. ‘ To introduce somewhat greater order into our conceptualization 
of our field. 

It is important for you to notice that there are some objectives that I did 
not mention, e.g,, 

. To provide a comprehensive bibliography of the field. 

. To provide historical overview of the field. 

To review a given area from the beginning of time. 

. To describe ’ ■ • part cuiar xu detai.^. 

. To list the major professional gatherings of the past year or the 
ones planned for the coming year. 

To list available hardware, software, or service. 

To teach novices. 

The poinr_ 1 am making is that there are many xinds of in: c rmatio.; tools and 
many kinds of needed information and it is important, in cainking about the 
Annual Review , not to confuse the tool we are trying to 1 ..e with the ones we 
are not trying to have, 1*11 come back to this point. 
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Atxnual Review 

The fiicichauisms for producing all annual reviews are quite similar: 

defij^^ Contents, select tbc authors and instruct them, edit the 

contJ^i-bui^i'^’^Sj and publish* However, the Annual Review has a very complex 
and mechauism. It differs from other annual reviews that I know 

about' in fi'V’e major -^ays: 

. yje Provide authors with bibliographies. The 1969 bibliography 

^Dtipained ovgr 5000 items. 

> provide Very detailed instructions and guidance and work 

■pal'd to avoij Or minimize interface problems betvreen chapters. 

• ha'y^e insp^tiuted a merciless review process to achieve tb 
^asired lev^i of quality. 

• ^pite of g^ii {;pe extra steps in our cycle, we manage to get 

AiiHual^^^ui^ out ia roughly half the time required by most 
pth^j- annual reviews. 

• Undertake empirical research to evaluate our Annual Re view . I 

of no Other annual review publication that can make that statement. 



Our bev'iew Ptocess is aS follows; Each year, the Editor and an 

Advi^'^ty ^®%nittee appointed by ASIS consider and agree on the content and 
orgaJ^^xa(/^*^b. of the next volume end on the authors to be invited. I 
init^'^te process^ drawing on (1) the framework provided by past ARIST volumes 
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(2) the amount of literature in areas of interest to the Annual Review , (3) the 
appraisals of the authors of the current volume regarding areas of growing or de- 
clining importance^ and (4) the date of the most recent coverage of particular 
areas in the Annual Review > Options are posed to the Advisory Gommitteej which 
indicates preferences and priorities for coverage and chapter authors • 

The chapter authors are provided with an Author^s Guide Indicating what is 
expected of them and providing guidance and advice on handling what for 
nearly everyone is an exceptionally difficult — at times almost impossible 
task. They are asked to screen literally hundred of potentially relevant items 
of recent — usually past-^year — literature; discard the outdated and unimportant; 
and then, in their chapter* describe and appraise progress in their area. 

They are asked — in fact, told — to make evaluative Judgments about both the 
technical work and the literature that describes it. 

The first draft submitted by each author undergoes exhaustive technical 
review by four members of the Advisory Committee and by fellow professionals 
with expertise in the particular area of the chapter* (I and my staff review 
and critique the draft at the same time.) Nearly all of the reviewers are 
past Annual Review chapter authors and understand both the content requirements and 
the authors^ problem. Their comments, suggestions, and criticisms — together 
with the marked-up manuscript are returned to me for analysis and integration 
into a 5“ to 15-page detailed feedback letter. The author is encouraged to 
attend to as much of the feedback as he considers valid and to make the 
corrections and changes indicated. He then rei^nrites the chapter and submits 
it for final editing before publication. 
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Impact of the Annual Review 

There are two bodies of evidence on the impact of the Annual Review. One 
stems from a study conducted in 1967, which was, to my knowledge, the first 
and only empirical study of any annual review. We contacted over 10,000 
individuals to determine who had actually used the Annual Review , who had 
seen or heard of it but not used it, and who had neither seen nor heard of 
it. . Separate questionnaires were sent out to samples from each of these 
groups to obtain more detailed data about their professional communication 
methods and, for the Annual Review users, the ways in which they had used 
the book. A second body of evidence is being developed now, as part of a 
survey of ASIS members. 

The results from both of the studies are very interesting and, in some cases, 
startling. One of the most interesting findings is that an annual review 

serves many, quite different purposes. Our original survey showed that the 

purposes for which the Annual Review was rated “Very Useful” were: 

. Keeping up with current work in peripheral areas of interest (35 percent) 

. Keeping up with current work in own areas of interest (35 percent) 

Learning about an area not within own professional specialt^^ (22 percent) 
Checking on particular projects or ideas (19 percent) 

A number of other uses were mentioned in the earlier survey as less important, 
including checking on the activities of individuals reading the original 
literature more selectively, identifying areas of information science that 
require further research, etc. We speculated about the relatively low value 
placed on the reference use of the Annual Review (i.e., “checking on particular 
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projects or ideas'*), and, at the time, I stated my belief that our figures 
underestimated what would come to be seen as the true reference value of the 
sei^ies. The new findings bear out this prediction. "Checking on particular 
projects or ideas,** ranked fourth in 1967, has jumped to first, with the 
percentages of users saying "Very Useful** going from 19 percent to 47 percent. 

It is Interesting that nearly all the other percentages are also up from 1967: 
**Keeping up with current work in own areas of interest" rose from 33 to 45 
percent; "Keeping up with current work in peripheral areas of interest** rose 
from 35 to 40 percent; and "learning about an area not within own professional 
specialty*' rose from 22 to 33 percent. 

The extent of use of a given volume is fairly stable. In 1967, the median 
number of chapters read \<ras four; in 1971, it is still four. In 1967, the 

median number of chapters skimmed was four; now it is about five. Incidentally?', 
many people believe that annual reviews are read from coV€ir to cover; that is 
why we have had suggestions for eliminating ^itll overlap among chapters and for 
integrating all chapter references in one large, combined list of references. 
However, the fact is that most people actually read only about four chapters, 
so it would be a disservice to the readers if the chapters were not reasonably 
self-contained and could not be read independently of one another. 

The two most read chapters in 1967 were the ones on library automation and 
on information needs and use; they are still the ones most read. The least 
read chapter in 1967 was the one on new hardware; last year, it was the one 
on computer technology. This suggests that the interest patterns of our 
readers may be fairly stable. 
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The data showed a number of interesting correlates of use of the Annual 
Review . The largest single group of users, in the 1967 survey, worked in 
industrial or business firms. Exactly the same finding — and even the same 

percentage, 36 — came out of our more recent survey. We also learned, in, 
the earlier survey, that users differ from nonusers in a number of ways, 
particularly on measures of professional activity, such as meetings attended, 
professional papers written and presented, etc. The users were uniformly higher 
on all measures of professional activity. Also, those users who had been in 
the field 10 years or more tended to use the Annual Review more than those who 
had been in the field for less time. This finding suggested that we were 
indeed hitting the target audience for the Annual Review . 

With respect to impact, I must say that measuring it is rather difficult. 

Ideally, one would like to have data com?piled over a period of time, rather than 
data collected at only a single point in time. Although we do not have a great 
deal of the most desirable kind of data, we do have some clear evidence of the 
impact of the series from several q^iestions asked in both surveys. For example, 
we asked; “Have you ever reexamined publications you had previously read, as a 
result of reading the Annual Review ? (Yes/No)." In 1967, 53 percent of the 
users answered ”yes;” in 1971, the *^es” figure was 68 percent. 

We also asked why readers examined any of the publications again. The 
primary reasons checked were to review forgotten details (30 percent in 1967 
and 38 percent now) , and to reevaluate a publication in light of an Annual 
Review author's comments (20 percent in 1967 and 45 percent now). 
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We asked: "After reading the Annual Review , have you tried to obtain some 

of the publications cited in it that you hadn't read before? Yes/No." In 
1967, 55 percent of the respondents said "yesl^' 1971, the figure was 

84 percent . 

We asked: ’*As a result of reading the Annual Review , have you tried to 

contact any of the authors mentioned for information regarding their current 
work? Yes/No.” In 1967, 10 percent of the users said ”y^s;” in 1971, the 
**yes’’ figure was 22 percent. 



We asked: "Has the Annual Review itself ever suggested to you specific ideas 

for future research projects or studies? Yes/No." In 1967, 45 percent of tfie 
users, said "yes;" in 1971, the "yes" figure was 50 percent. 



These findings strongly suggested that the Annual Review has had — and continues 
to have — a strong positive impact on professional communication. One of the 
interesting aspects of the earlier study, which we have not yet explored in the 
more recent study, was the lack of any evidence that use of the Annual Review 
was associated with decreased use of the primary literature. This is consistent 
with the findings of a "synergistic" effect among information channels: 
information channels interact and mutually reinforce one another. It would 
appear that the Annual Review has this kind of effect for many professionals. 
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Responses to the Issues, Criticisms, and Recommendations 

Having provided this context for you, I will now address the three major sets 
of criticisms and suggestions I mentioned at the outset. 

The **Absence** of Critical Evaluation . I agree with the feeling that the review 
should be something more than a critical annotated bibliography. I disagree 
on several things. I do not believe it is fair to characterize the approach 
taken by Annual Review authors as "uncritical,** although in some cases it is 
true. I also do not believe that the items discussed in the Annual Review 
receive equal treatment and equal weight or that the reader cannot obtain an 
Impression of what works are of high or low quality. There are many clues to 
comparative value, including the length of discussion, prominence in a given 
section, and presence of evaluative comments, of which I believe there are 
many. However, one thing I will concede is that the reader cannot readily 
identify those works> of no quality. That is the part of the literature that 
I tell the author to throw in the wastebasket before he writes the chapter. 

Although it may not always be clear to our readers, it is a firm policy of 
the Annual Reviexv — and always has been — to encourage critical evaluation. 

So why isn't there more of it? One reason is that we discourage evaluation 
that does not have a firm basis. We discourage essays filled only with 
opinion. If a review chapter is not firmly grounded, it is — in the words of 
our advisors — **just one more damn thing to read.** It is definitely not our 
purpose in the Annual Review to create one more damn thing to read, but 
rather to remove part of the necessity to find and read all those things. 
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A second reason for the lack of critical evaluation is the author’s own 
self-control, based on his awareness of the powerful platform that the Annual 
Review provides. A few authors have been a little careless with the 
privilege: one wanted to spend several pages lambasting someone’s poor work. 

I dissuaded him, pointing out that the time-honored way for an annual review 
to handle nonsense is to leave it out, not give it precious space that ought 
to be used for really important and useful work. 



I believe that the si esc: ->n we abandon the pr '^sent Annual Review format and 
go to a combination of state-of-the-art reports and an annotated bibliography 
stems from a fundamental .isunder standing of the purpose of annual reviews. 

An annual review is: 

• Not a state-of-the-art report 

• Not an annotated bibliography 
. Not a critical analysis 

. Not a list of suggested readings or a buying list 

• Not a current awareness device 

• Not a "who’s who" or "who’s doing what" reference tool 
Not a primer 

. Not a textbook 



It is clear from our data that an annual review is , in fact , a particular and 
unique art form that is something of all of these, without being optimized for 
any particular one. It serves many quite different uses for people of different 
professional levels and with different responsibilities. I agree that there 
should be more and better state-of-the-art studies and better annotated 
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critical bibliographies, but neither of these is an appropriate substitute 
for an annual review. 

In our recent survey, we put the question directly: "Does the field of 

information science need more state- ’ art reports (monographs covering 
each topic in full) than are usually pi Ixshsl ea 'h year?" Sixty— seven percent 
of the users said "yes," but only one ox: rf ten , arsons thought that the 
state-of-the-art reports should replace inua Review . Nine out of ten 

persons thought that they should be establii ad as a separate type of 
P^^l^^-^bion. It may be appropriate (if pai: i al) tc see the Annual Review 

not being a good annual revi-v, , but it hardly seems appropriate 
to criticize it because it is not something different. • 

OYotemphas is on Technology . It would be tempting to make a copout and 

say that the content is selected and agreed to by the ASIS Advisory Committee. 
Tnxs is partly true, of course, but the real answer is that the Annual Review 
wais intended to serve a particular target audience— —the membership of ASIS— — 
plus professionals in fields closely related to ours. Most ASIS manbers are 
not .engaged in science: they are engaged in information technology, information 

applications, and information service. That was a fact of life in 1964, when I 
first proposed the Annual Review and insisted that we make the full title Annual 
Review of Information Science and Technology . It is still a fact of life. The 
i^nual Re view includes relatively little science because our menibership, and 
that of related professional associations, are involved readily little in 
scientific activities. 
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The recommendation was made that we give theory a prominent review, when and 
where it exists. Almost every year, for the past 7 years - have asked our 
advisors about handling information science theory. I asKed them agaxn this 
year whether we should have a chapter on theory, or cove heory in 
an occasional chapter, or cover theory in the context of o. ler chapters. The 
tnajority view has always been the same: cover theory in the context of other 

chapters. 

The recommendation that we pay attention to the social and philosophical con- 
text of our field is one to which 1 have already given special attention. I 
recently suggested to our Advisors that we develop a new chapter on the "Social 
Implications of Information Science." After they agreed, I spent 2 months 
trying to find a qualified and interested author. In March I had to abandon 
the search as far as Volume 7 is concerned, but I would welcome suggestions 
regarding qualified author candidates for a later volume. 

inclusion of the, "Mrona" Literature . Of all the assertions and recommendations 
in the appraisal, the most surprising to me is the assertion tnat very few 
technical reports are worthwhile and the recommendation that the ^ ual Revi^ 
essentially omit such reports from its coverage. In response, I must point 
out that the Annual Review authors, working both from their own literature 
sources and from ASIS-provlded bibliographies containing up to 5000 current 
items of potential interest annually, have selected the items that they deem 
worthy of discussion. Some authors have, in fact, examined as many as 700 books, 
journal articles, reports, and other printed items, ending up with 100 to 200 
:R|c items that they consider particularly worthv of discussion or mention. To 
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sug03st that thaae chapter authara are enable to determine what ia worthwhile 
and that one ahonld substitute for their professional Judgment the simple 
criterion "was It published in a journal" is incredible to me. If the chapter 
authors, after all their work, cannot Judge what is useful Information, who can? 

The argument can he made that ^ worthwhile technical reports eventually o„d 

up in Journals anyhow, while this may conceivably he true for fields such as 

physics. Chemistry, and medicine, which have a well-established Journal struc- 

it IS decidedly, not true for information science. Many of the Individuals 

that I personally, regard as "real pros" in information do not publish a great 

deal, not only because they are heavily engaged in a continuing stream of new 

project work (both research and applied) but because, the technical reports 

they do vn:ite are not Journal-sited, bet's face it: Journals publish Journal- 

site packages. For every good report that finds its way into hot type, I 

believe that there are several others that do not. The point, of course, 

not that technical reports as a class are better than published articles, 

that articles are better than reports. It is that evaluative Judgments should 

be made by professioa^ on the basis of jdm. is said, rather than on the medium 
in which it is said. 
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In our recent survey, „e asked ASIS members directly: "Ho„ useful would it be 

to ^ technical report literature from Annuai Review coverage, confining it 
largely to Journal literature and books?" Five choices were posed: 

( ) Much less useful 

^ ^ Somewhat less useful 

^ ^ Equally useful 

^ ^ Somewhat more useful 

( ^ Much more useful 
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Of the users, 87 answered either "Much less useful" (55) or "Sc what i:iss 
uscuzul" (32) V, Only 3 said that this would be "Somewhat more ul ^ _;1" or 
"Much more useful." There is no question regara^ng the attitude of users of 
the Annual Review s technical reports should not be eliminated f om our 
coverage. 

Concluding Comments 

In this talk I have necessarily dwelt on the points of disagreement with the 
appraisal paper. I must stress the fact that the appraisal paper^ ^Iso contains 
a number of points of fair and valid criticism, as well as suggestions well 
worth considering. I am grateful to Tefko for calling them to m attention 
and I, in turn, have promised to call them to the attention of the ASIS Advisory^ 
Conmiittee. I also welcome comments and suggestions from other colleague.^? . All 
of us share an interest in the success of the Annual Review and in its service 
to the Information science profession, and we can fulfill that interest best 
by intelligent dialogue along the lines Tefko has initiated- 

I think it would be very unfortunate for the Annual Review if I were to become 
(or remain) complacent about needed improvements to the series* I*m grateful to 
the organizers of your meeting not only for the building part of it around the 
Annual Review but for providing me with an opportunity to bring several 
important issues to your attention and to stimulate your thinking about the 
future of the Annual Review series. 




